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ABSTRACT 


This thesis was a study which deals with two basics concepts in human 
decision making. The first is the role of information relevance, specifically the 
adverse effects of irrelevant information on decision quality. The second key 
concept was cognitive feedback and its value for supporting decision making. The 
thesis was designed to research the effectiveness of cognitive feedback in reducing 
the adverse effects of irrelevant information. The experiment tested the Lens 
Model indices: achievement, consistency and matching in task conditions of high 
and low predictability. Subjects were divided into blocks which differed in the 
availability of cognitive feedback and predictability conditions. The results of the 
experiment showed the subjects performed better in all Lens Model indices in the 
cognitive feedback condition. Subjects also had superior performance across all 
Lens Model indices in the high predictability condition. This thesis was intended 
to contribute to the research in the subject of human decision making. The results 


were of importance in support of future design of decision support systems. 
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I. Introduction 


A. Research Context 
1. Use of feedback on decision making 

Researchers in several fields have studied the effects 
of feedback on decision making. From the fields of psychology 
and psychiatry, research has studied the way in which 
cognitive feedback is used by an individual to understand 
his/her environment (Doherty and Balzer, 1988). Researchers 
in the field of marketing have been interested in how "good" 
decision makers depend on the use of cognitive feedback 
(Jacoby, Mazursky, Troutman, and Kuss, 1984). Research has 
shown that some forms of feedback can have an effect on the 
quality of decision making. From this research, and others, 
there have emerged several theories which support this thesis. 
These topics will be discussed in detail in Chapter II. 

2. Effects of irrelevant information in decision 

making 

The effect of irrelevant information on decision 
making has been a subject of continuing attention in the field 
of social and clinical psychology. Research has been directed 
at the effects of information relevance on decision making 


(Streufert S., 1973), reducing the effects of irrelevant 


information on experienced decision makers (Gaeth and 
Shanteau, 1984), and the perception of information relevance 
(Streufert and Streufert, 1970). One well accepted theory 
concerning information relevance is that the presence of 
irrelevant information is detrimental to decision making. The 
concept of information relevance is of key importance to this 
thesis. Research and theories pertaining to information 
relevance will be discussed in Chapter II. 

The research proposed here addresses the two points 
made above - (a) given that the presence of irrelevant 
information has an effect on decision making, (b) can the use 
of feedback reduce the effects of irrelevant information. In 
this study we will examine the effects of irrelevant 
information in task environments characterized by different 
degrees of predictability. We will draw conclusions as to the 
effectiveness of feedback as a method of reducing the effects 
of irrelevant information. 

3. Relevance to information systems research 

The relevance of this study to research to Information 
Systems (IS) is in the domain of Decision Support Systems 
(DSS). With the rapid growth of demand for both specialized 
and off-the-shelf DSS, emphasis is being placed on the quality 
of design.  DSS provide users with information in order to 
enhance the quality of their decisions. Not all information 


displayed to the user at one time is relevant. If the 


decision maker is given the opportunity to study his/her own 
decision rules through the use of some form of cognitive 
feedback, he can improve his/her decision rules. Not all DSS 
of today provide CFB to the user. This research will help to 
determine if such a capability can improve the quality of the 


DSS. 


B. Problem domain 
In this section, we outline the elements of the problem 
domain, thereby constituting the basis for the research 
question - the notion of cognitive feedback as an aid to 
decision making in environments characterized by the presence 
of irrelevant information. 
1. Irrelevant information 
The effects of irrelevant information on decision 
making have been studied extensively. The consensus from this 
research yields two basic premises that are essential to this 
study. First, the presence of irrelevant information is a 
detriment to decision making (Gaeth and Shanteau, 1984). 
Second, the adverse effects of irrelevant information are 
greater in complex environments than in relatively simple 
environments (Streufert and Streufert, 1973). 
2. Cognitive feedback 
The term feedback describes an environment that 
returns some measure of the output of a system back to the 


system which produced the output. Feedback then allows a 


person to compare his/her present state to an ideal state, to 


adjust itself in light of that comparison (Doherty and Balzer, 


1988). Feedback can be provided in two forms - outcome and 
cognitive. Outcome feedback deals with the accuracy of a 
response. Cognitive feedback provides information as to how 


that response was generated. Cognitive feedback is the return 
of some measure of the output of a person's cognitive 
processes, to help that person come to terms with his/her 
environment. 

Cognitive feedback can take three forms in an 
experimental setting: 

(a) information about the relationships between cues 
and criterion, i.e. information about the task: 

(b) information about relationships between cues and 
the person’s inference, i.e. information about the person’s 
cognitive state, sometimes referred to as insight; 

(c) information about relationships between 
cognitions and distal objects. This category comprises 
indices of "functional validity" information (Doherty and 
Balzar, 1988). 

3. Research question 

Given that the presence of irrelevant information 

adversely affects task performance, can the use of cognitive 


feedback improve the quality of decisions? 


C. Organization of the thesis 

The discussion proceeds as follows. Chapter II reviews 
the research in the effects of irrelevant information and the 
use of feedback in task situations. Chapter III describes the 
experimental setting and chapter IV discusses the data 


analysis and results. Chapter V summarizes and draws 


conclusions. 


II. RESEARCH 


A. The Research Question - A Conceptual Framework 

The research question deals with two basic theoretical 
areas: information relevance and cognitive feedback. The 
purpose of this chapter is to lay the groundwork in these two 
research areas, giving logical support for the hypotheses to 
be made later in Chapter III. Important to both of these 
research areas is the topic of human judgment. 

l. Human Judgment 

Understanding human judgment is important in the 

context of this research. Changes in human judgment are what 
we hope to effect by manipulation of the dependent variables. 
A background discussion of human judgment theories is included 
in section  A.1. The mathematical and experimental 
representation of human judgment is provided by the lens model 
(Dudycha and Naylor, 1966). The lens model and issues 
surrounding linear models of human judgment will be discussed 
in section B.1. 

2. Information Relevance 

Research in the area of information relevance is 

extensive (Streufert, 1973; Gaeth and Shanteau, 1981, 1984; 


Adelman, 1981). It is commonly accepted that irrelevant 


information serves as a detriment to decision making. The 
effects of information relevance vary under different 
circumstances. Section C will discuss the elements of 
information relevance as it pertains to this research. 

3. The Research Problem 

The research problem can be framed by the following 
statements: 

(1) Individuals make models of decision processes. 

(ii) Irrelevant information serves as a detriment to 
successful development and use of a decision model. 

(iii) Cognitive feedback has been proven successful in 
improving decision making, by providing decision makers a 
better insight into their model development and usage 
strategy. 

(iv) So, given that cognitive feedback improves 
decision making, can it be useful in overcoming the 


detrimental effects of irrelevant information? 


B. Human Judgment 

According to Brehmer, knowledge is a relation between two 
systems. One system in this relationship is, for the purpose 
of studying human judgment, a person and the other is some 
portion of the environment. A person who is believed to 
"know" a great deal is someone who understands relationships 
with his/her environment better that someone who understands 


these relationships less. These relationships are believed to 


be  probabalistic. Thus, perfect knowledge is infeasible, 
only knowledge where the person has a high probability that 
what he believes he knows, is actually the case. 

It has been the focus of many researchers in experimental 
and clinical psychology to study the way in which humans seek 
relationships between themselves and their environment 
(Brunswik, Brehmer, Doherty and Balzer, et al). Doherty and 
Balzer referred to knowledge representation as a complex set 
of relations called a policy. Policies are sampled from the 
stimuli and then analyzed for their relevance. The relevant 
components are then tried for their usefulness, either one by 
one or in some combination (Doherty and Balzer, 1980). 

The use of linear models was first suggested by Brunswik 
(1944) for studies of perception. A linear model suggests 
that there is a probablistic and functional relationship 
between an individual and his environment, and that the 
functions can be described and measured. These methods are 
used to test a series of hypotheses about the nature of the 
judgment process, hypotheses about the nature of cue weights, 
function forms, combination rules, and predictability 
(Brehmer, 1979). The name of the model developed by Brunswik 
was the Lens Model. 

1. The Lens Model 

The Lens Model is a linear regression model developed 


by Brunswik for assessing the dynamics of human choice 


behavior in a probablistic environment. The following text 

from Dudycha and Naylor describes well the components of the 

Lens Model: 
The three basic elements to the model are the cues or 
stimulus dimensions (X, .... X,), the correct response or 
answer (Y), and the observed response of the individual 
(Y): Any choice or decision situation (trial) must of 
necessity include these three elements. Given many such 
decision trials it is possible to determine the 
statistical relevance of a cue X, as a predictor of the 
criterion Y, by computing the zero-order correlation r.i 
over trials. This true cue validity, when squared, can be 
interpreted as an index of the diagnostic power of that 
cue as a source of information for predicting the correct 
state of nature. 

Also from Dudycha and Naylor are the following values 
used in the model: 

re = the correlation between the true or observed 
criterion and the predicted criterion. 

r, = the index of subject consistency, or the degree 
to which the subject consistently utilized his strategy as 
defined by his multiple regression equation. 

ru = the correlation which denotes subject 
achievement. It measures the degree of agreement between the 
criterion values and responses of the subject over n 
observations. 

r, = the correlation between the two sets of n 
predicted values, which reflects the degree to which the 
regression (policy) equation of the subject "matches" the 


regression (optimal) equation of the ecology. 


ree = the correlation which reflects the degree that 


the optimal equation of the environment can predict the actual 
responses of the subject. 

r,, = the correlation which reflects the degree to 
which the policy equation of the subject can predict the true 
criterion. (The above variables and definitions taken from 
Dudycha and Naylor, 1966). 

The Lens Model is an accepted method for interpreting 
subject responses in single or multiple cue probablistic 
learning environments with linear tasks. Tucker further 
developed the model for tasks in which the judgment is a 
linear additive function to yield the following relationship: 

ra = GRR, 
R, represents the predictability of the environment, R, 
represents the consistency of the subject, and G is the 
correlation between the predictions derived from the linear 
model of the environment with the linear model of the subject 


(matching index). Thus, r, is called the achievement index. 


C. Information Relevance 

Relevant information is viewed as any information to which 
a meaningful task-oriented response is possible (Streufert, 
1973). Thus, information which is not useful in the 
generation of meaningful responses is irrelevant. In an 
experimental setting, as well as in nature, irrelevant 
information is often mixed with relevant information. 


In terms of the lens model, irrelevant information is a 
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cue which has a low r.. A cue with little or no diagnostic 
power (diagnostic power is described in the previous section) 
is mostly or entirely irrelevant. 

Study in information relevance has been extensive. Most 
data suggest that increases in irrelevant information decrease 
performance to criterion (Streufert, 1973; Streufert and 
Streufert,1971; Gaeth and Shanteau, 1984; Ettenson and 
Shanteau, 1987; et al). This premise is well accepted as a 
starting point for further research in the area of information 
relevance. 

Gaeth and Shanteau (1984) sought to discover if this 
influence extended to experienced decision makers. In their 
study, the decision makers were trained agricultural students. 
They were asked to describe soil texture. Irrelevant to this 
determination is moisture in the soil. One important result 
of the experiment is benefit of training in reducing the 
adverse effects of irrelevant information. If this influence 
extends to experienced decision makers, then one reasonable 
approach to improving judgmental skills would be to reduce the 
effects of irrelevant information (Gaeth and Shanteau, 1984). 

1. Effects of Irrelevant Information 

The effects of irrelevant information are known to 
vary in different task environments. For example, adding 
irrelevant information in simpler environments usually does 


not have as great an effect in producing decrements in task 
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performance (Streufert and Streufert, 1970). A subject can 
perform well in a task environment where irrelevant 
information is present as long as he can sort out relevant and 
irrelevant information. In complex environments it is more 
difficult to differentiate between relevant and irrelevant 
information. 

Streufert (1973) performed a study demonstrating the 
decreases in performance levels previously believed to be the 
effect of load increases were actually a function of 
information relevance. 

2. Effectiveness of Training 

Gaeth and Shanteau’s research using training is of 
hallmark importance to research in information relevance. The 
study utilized experienced agricultural judges in a soil 
sampling task. Ina pretest, irrelevant information was shown 
to influence the decisions of the judges. This pretest is of 
some consequence, for it shows that even experts are subject 
to the effects of information irrelevance. Subsequent 
training, in two different forms: lecture training and 
interactive training, was shown to improve accuracy and reduce 
the influence of irrelevance. Gaeth and Shanteau’s research 
utilizing training is of hallmark importance as it is the 
first substantial work investigating the potential for 
reducing the adverse effects of irrelevant information. 


Additionally, it is of great significance that the study was 
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able to differentiate between improvements in accuracy and 
decreases in the effects of irrelevant information, proving 
that the two are mutually exclusive. A follow-up study showed 
that the benefit from training was shown to extend for a year 


after the training was held. 


D. Feedback 
1. Outcome Feedback 
The procedure of informing the subject in an 
experiment of the correct value in a task situation (Y,) 


immediately after that subject produces a response  (Y,) 


defines outcome feedback (OFB) (Doherty and Balzer, 
1988). Research has attempted to correlate improvements in 
decision quality based of the use of OFB. It is commonly 


accepted that OFB alone is of little assistance to decision 
makers learning complex inference tasks. OFB can in some 
cases serve as a detriment to decision making because it 
encourages departure from linearity (Lindell, 1976). This 
departure from linearity is especially likely when the task 
environment is less than complete predictability. 
Brehmer (1980) explains the reasons why OFB is not 

useful for enhancing decision making: 

Confirmation will, of course, not teach the subjects about 

the actual validity of their hypothesis; it will only tell 


which hypotheses work, although the reason why the 
hypotheses work may be very different from what the 


subject thinks... When we have to learn from outcomes, it 
may in fact, be almost impossible to discover that one 
really does not know anything. This is especially true 
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when the concepts are very complex in the sense that each 
instance contains many dimensions. 


One effect seen when OFB is used alone is when a 
subject is given OFB demonstrating that his response is 
incorrect, he will often abandon the policy which was 
employed to produce the response, even if it was the correct 
policy. Repeated failure to get the "right" answer may cause 
the subject to abandon the idea that there is a policy at all 
and he may resort to guessing (Doherty and Balzer, 1988). 

Outcome feedback in a less than perfectly predictable 
task apparently acts to confuse subjects and lower their 
consistency. Achievement is also adversely affected, because 
of this lowered consistency (Schmitt, Coyle, and Saari, 1977). 
It follows that outcome feedback in task environments of low 
predictability is not useful for policy revision. 

2. Defining cognitive feedback 

Cognitive feedback is described as the return of some 
measure of the output of a person’s cognitive processes, to 
help that person come to terms with the environment. 
Cognitive feedback provides the person with information 
describing the relationships: 

(a) between cues and the criterion (Task Information 
1 

(b) between cues and the person's inference, i.e. 
information about the person's policy (Cognitive Information; 


CI) and/or 
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(c) between cognitions and the distal objects. This 
category comprises indices of functional validity (Doherty and 
Balzer,1988). The third category, above, is added to parallel 
the lens model accurately. In the lens model, TI includes the 
indices on the ecological side of the lens. CI includes all 
information on the subjects side. Functional validity refers 
to all three measures r,, G, and C described in section B.1. 

In terms of the Lens Model, TI is represented by R, 
or R,*?, ecological validities (earlier called r., Or diagnostic 
power, see section B.1) or function forms relating the 
Criterion to the cues. CI is represented by the values R, or 
R,* in the Lens Model as correlation indices of predictability 
of the subject (consistency). Another form of CI are the 
usage coefficients (e.g., r,,; the subject's decision policy) 
and function forms relating the judgment to the cues. 

3. Effectiveness of Cognitive Feedback 

Cognitive feedback has been used in numerous 
experiments to produce changes in decision making policy. 
Cognitively oriented feedback results in higher levels of 
achievement than outcome feedback over different task 
properties (Adelman, 1981). Other studies have demonstrated 
the superiority of CFB to no information, as evidenced by 
policy change between blocks after providing CFB. Research 
has been conducted to determine which component of CFB (TI, 


CI, or FVI) is most effective in making change. The bulk of 
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research on the subject shows that TI alone is sufficient to 
facilitate change. There has been research investigating the 
use of CI alone, and TI + CI. There is not adequate research 
to make assumptions about differential effects. All lens 
model indices seem to be influenced by TI and TI + CI with R, 
being more sensitive (Doherty and Balzer, 1988). 

The use of CFB has been found to be related to task 
predictability. The use cognitive feedback under conditions 
of extremely high task congruence and predictability has not 
been shown to produce higher accuracy than outcome feedback 
alone. Schmitt et al. (1977) found that as task 
predictability decreased, achievement with cognitive feedback 
became significantly higher than that with outcome feedback 
(Adelman, 1981). Predictability is varied in experimental 
settings to manipulate task complexity (Steinman, 1976). 

One finding of double-systems studies pertains to the 
effects of task predictability on the cognitive systems of the 
subjects. Results obtained in a variety of circumstances show 
that the consistency of a cognitive system varies with the 
redictability of the task, the lower the consistency of the 
cognitive system (Brehmer 1979). This effect is related to 
the theory that policy formation and utilization is 


probablistic. 
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E. Framing the Research Question 

The previous sections are valuable for demonstrating the 
need for and validity of research in the proposed area. In 
the section on irrelevant information it was shown that a 
measurable portion of difficulty experienced by decision 
makers in complex tasks was due to the effects of irrelevant 
information. Streufert went one step further in showing that 
Similar losses in accuracy previously blamed on load were 
actually a result of irrelevant information. Some research 
has begun to discover means for recouping or avoiding these 
losses. Gaeth and Shanteau have had success using training as 
a means to this end. They further point out that research in 
irrelevant information is abundant, while research into 
correcting for it is overdue (Gaeth and Shanteau, 1981). 

The use of cognitive feedback has been proven to be an 
effective means of improving performance in many different 
task oriented environments. The bulk of the research 
indicates that the more complex and predictable the task is, 
the more profound the benefit is when cognitive feedback is 
utilized. It is reasonable, then, to believe that it will 
also have a measurable effect on decision making in task 
environments subject to the presence of irrelevant 
information. 

These are the fundamental principles of the study. It is 
clear from these themes that not only is the theoretical 


foundation for the study valid, but there is evidence that the 
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results of the research will benefit all pertinent areas of 
study. The research question is further developed Chapter 
III, which describes the experimental methodology, and Chapter 
IV, which discussed the same research but in relation to the 


outcome of the experiment. 
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TII. METHOD 


A. Scope 

This chapter describes the experimental design used in the 
research. The experiment was divided into three phases: 
training, experiment, and debriefing. The rationale for the 
training is explained in the previous chapter. Its results 
will be described in this chapter in section G. Section I 
discusses the debriefing phase, and section J is a summary of 


this chapter. 


B. Hypotheses 

The first hypothesis is discussed in terms of the lens 
model index for achievement. All hypotheses were formulated 
on the individual level. 

Be: Subjects receiving CFB will attain higher decision 
quality than those receiving only OFB. 

H,: Subjects will perform better in task environments of 
high predictability than those in environments of low 
predictability. 

1. Description of Hypotheses 

Based on arguments discussed previously in Chapter II, 
we postulate that performance will improve in the cognitive 


feedback condition. Performance in this sense means that the 
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subject’s responses will more closely resemble the committee 
score. Similarly, the subject's policy will resemble the 
committee policy. 

Section C.1 of Chapter II reviews Streufert's findings 
that the presence of irrelevant information is more 
detrimental to decision makers in environments of higher task 
difficulty. It is proposed in H, that this effect will also 


hold true to differences in task predictability. 


C. Experimental design 


The purpose of this section is to describe the 


experimental design of the experiment. The experiment has 
three phases: training phase, experimental phase, and 
debriefing. 


1. Description of design 

The experimental design has two components - between 
subjects effects and within subjects effects. The design 
chosen to accomplish this was a 2 (OFB vs CFB) X 2 (High vs 
Low Predictability) X 3 (Block sequence) factorial design. 

The between-subjects effect is the difference in 
performance between those subjects given cognitive feedback 
and those only given outcome feedback. Another between- 
subject effect is the difference in performance between those 
subjects in the high predictability task environment and those 
in the low predictability environment. The between subjects 


design is described by Figure 1 below. 
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As well as determining if systematic differences exist 





Figure 1 


among experimental conditions, this study also seeks to study 
the effect within each condition over time. This within- 
subjects design involves measurements that occur over time. 
Time of response will be measured for all subjects to see if 
learning was occurring with each subject. 

The within-subjects effect of irrelevant information 
was operationalized by varying which cue (of three) was 
irrelevant for each block. The order that each subject 
receives the blocks of trials was then operationalized using 
a Latin Squares design following the procedure in given in 
Kirk (1982). 

The following figure demonstrates the separation of 
within-subjects. 


Group 1 Group 2 Group 3 


Task order: Cue 2 Cue 3 Cue 1 
Cue 1 Cue 2 Cue 3 
Cue 3 Cue 1 Cue 2 


Therefore, group 1 gets the order 2, 1, 3; group 2 gets the 
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order 3, 2, 1; and group 3 gets the order 1, 3, 2. For 
example, subjects in group 1 will receive a task sequence of 
the following order: in block 1, cue 2 was irrelevant; in 
block 2, cue 1 was irrelevant; and in block three, cue three 
waS irrelevant. 
2. Experimental setting 

The sessions were conducted in a closed room in the 
presence of the experimenter. The training and experimental 
phases were conducted on an IBM compatible personal computer. 
Brief instructions were provided by the experimenter at the 
beginning of the training phase. From that point, the subject 
used the instructions provided and on-screen help provided by 
the software to progress through the experiment. The software 
was written by the experimenters. The debriefing phase 
involved filling out a brief questionnaire attached to the 


instruction booklet. 


D. Subjects 
l. Choice of subjects 

The experiment was conducted with sixty subjects at 
the Naval Postgraduate School. Each subject participated 
individually. Each participant was assigned to one of four 
cells of subjects. These four blocks are described in Figure 
19 Subjects were assigned to one of four between-subjects 
conditions and one of three within-subjects conditions. 


Randomization was ensured in both of these assignment 


22 


procedures. 

Information of demographics and task specific factors 
that may impinge on the results of the experiment was 
collected. Demographic factors studied for interaction in the 
experiment include: age, sex, full-time work experience, 
familiarity with computers, and time lapse since undergraduate 
study. The task-specific factor was related to whether the 
subject had any previous experience in the task. 

2. The use of students as subjects 

One limitation of the study centers around the use of 
students in a laboratory environment. All participants had a 
minimum of four years management experiment. The issue is the 
extent to which it is possible to make a reasonable comparison 
between the subjects of the experiment and real-life decision 
makers. 

Previous research using graduate students as 
surrogates for managers failed to find any significant 
difference between the two groups in making production 
Scheduling decisions. It is reasonable to assume that 
graduate students are acceptable as representative of decision 
makers. 

It is difficult to claim external validity for 
laboratory studies. The bulk of the research conducted in 
support of the premises which frame the research question is 


conducted in a laboratory setting. The similarities between 
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other research in the area of irrelevant research and this 
research indicates that it is no less suitable for 


generalization to real-life. 


E. Task 

The following section describes in detail the experimental 
task (see Appendix). Also described are the different task 
environments in which subjects groups performed. 

l. Description 

The instruction booklet given to each subject varied 
based on the task. Subjects allowed to use cognitive feedback 
were — additional instructions describing the different 
types of feedback at their disposal and the function keys to 
access them. The debriefing questionnaire was also different 
for the two different types of subjects. The only difference 
in the debriefing questions pertained to how useful the 
subject found the types of feedback were in making their 
decision rules. 

The task presented to the subject was a candidate 
screening scenario. The subject was given a set of three cues 
representing three scores of a potential job applicant. The 
three scores ranged from 1 to 9. Each of the three scores 
represented an applicants score or rating on each of three 
variables or conditions: experience relevant to the position 
(variable 1), general abilities test score (variable 2), and 


interviewer rating of management abilities (variable 3). 
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Based on these ratings the subject was then required to award 
an overall rating of these candidates on a scale of 1 to 9. 
In the training phase these sets of three cues were 
presented to the subject one set at a time. The subject gave 
his response and the committee score was then shown. In the 
experimental phase, the sets of cues were blocked into groups 
of 26 applicants. This allowed for the subject to see the 
effects of his policy over several cases. There was no 


irrelevant cue in the training phase. 


F. Feedback 
The use of cognitive feedback was provided to half of the 
subjects in the experiment. Feedback was explained to the 
subject and made available. It was given to the subject only 
on request, and was available in a variety of forms. The 
various types of feedback are described in the next sections. 
1. Subject policy feedback 
The purpose of policy feedback is to allow the subject 
to see the policy he is using in order to refine and improve 
upon it. This type of feedback was presented to the subject 
in the form of a bar graph. Each cue was represented on the 
bar graph in a different color or shade (see Figures 2 and 3 
below). The size of each shade was determined by the weight 
that the subject was placing on this variable in arriving at 
his overall score for the applicant. If the cue has negative 


value, 1.e. a high score in a variable would lower the overall 
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Figure 2 


Score, it was shown in appropriate size for its negative 
weight, but below the positive values. 

In previous research conducted on representation of 
cognitive feedback, it was found that visual information was 
the most useful to the subject (Doherty and Balzer, 1988). 
Hammond (1971) stressed the importance of giving the subject 

a picture of his cognitive processes. For this reason, the 
graphical representation shown in the above figure was used. 
Each of the three cues was clearly distinguishable, making the 
figure clear and easy to understand. 

Cue weights were calculated as follows: 

(a) Beta weights were calculated from a multiple 
regression of cue values and the subject's estimates. 

(b) The weights were then transformed to represent 


percentages of the sum of the squared weights. Thus, beta 
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Figure 3 


values of 0.4, 0.8, and 0.2 were shown as 0.2, 0.75, and 0.05. 

(c) Transformed weights on the three cues were then 
displayed as a horizontal stacked bar, on a 1-100 scale (i.e., 
the transformed weights added up to 100). 

This form of feedback is described as a type of 
Cognitive Information (CI) in Chapter II. This type of 
feedback provides the subject with the function forms that 
relate his/her judgment to the cues. The other type of 
cognitive information discussed in Chapter II is consistency 
information. 

2. Consistency feedback 

Consistency feedback gives the subject an indication 
as to how well he was conforming to his decision policy. It 
may not be inherently obvious that different sets of responses 


can generate the same policy, but one set of responses may be 
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much more tightly grouped around the committee scores. Thus, 
the consistency feedback will provide the subject an 
indication as to how consistent he was with his mental model. 
It was designed also to give some assistance in improving his 
consistency. Both of these goals were achieved by providing 
the subject with consistency scores on individual cases. In 
other words, the subject will be given the values that he 
would have generated if he conformed strictly to his decision 
rule. 

Consistency feedback is a form of CFB called cognitive 
information (CI) by Doherty and Balzer (see section 2.4.) In 
terms of the lens model, consistency information is R, or R’,. 
It can be expressed in terms of the correlation indices of 
predictability of the subject. The use of consistency 
"scores" was chosen to make it easy for the subject to 
understand and use. 

Consistency scores were calculated as follows: 

(a) First beta weights were derived from a multiple 
regression of cue values and the subject’s estimates. 

(b) Criterion values were computed by multiplying cue 
values with respective beta weights. 

3. Committee decision policy 

The purpose of this type of feedback was to provide 

the subject with a reference to model his policy after. It 


was stated in the instructions that the purpose is to model 
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the decisions of the committee. This type of feedback was 
presented to the subject in the same manner that his own 
policy was presented, i.e. through the use of a bar graph. As 
before, each cue was given a section of the bar graph 
proportional to the weight that the committee places on that 
cue. In every set of cues, two of the three were weighted 
equally and the third was irrelevant. 

This type of feedback is called task information (TI) 
by Doherty and Balzer (see section 2.4). It is accepted as 
the most useful in effecting a change in behavior. This 
information provides the subject the function forms relating 
the criterion to the cues. 

4. Outcome feedback 

The purpose of this type of feedback was to provide 
the subject some indication as to his performance, also known 
as the committee score. This form of feedback was presented 
to the subject as a single number value. This value was 
calculated by summing the result of multiplying each cue by 
its beta weight. This type of feedback gives the subject an 
indication of his achievement. This value is denoted as R, in 
the lens model. 

5. Combined feedback 

The purpose of this feedback was to allow the subject 

to view two different types of feedback at one time. For 


example, the subject can ask to see the committee policy and 
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his own policy side by side. This was a convenience provided 

the subject to allow him the best conditions for refining his 

policy. The types of combined feedback allowed to the subject 

are: committee and subject policy rules, information on 
subject decision rule and consistency. 
6. Multiple requests for feedback 

The subject was allowed to request feedback many 

times. The only restraint on this was that once a subject has 

requested outcome feedback, he may no longer make revisions to 


those scores for which the committee score was given. 


G. Training 
l. Basis for training 

The purpose of training in the experiment was ensure 
that all of the subjects begin the experiment with the same 
knowledge of the task. Due to the context of the experiment, 
some subjects may come into the experiment with some 
preconceived ideas about which variables should have greater 
weights in their policy, i.e. some may feel that the interview 
score 1S more important than job experience. In the training 
phase, all cues were weighted equally (there was no 
irrelevant cue). At the end of the training phase, all 
subjects should have adequate knowledge of the task to perform 


well. 
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2. Training procedure 

During the training phase, all subjects received the 
same set of data. Each subject was asked to respond to 70 
cases. The subject was required to give each candidate an 
overall score from 1 to 9. This was followed by the actual 
committee score. The cues, subject response, and committee 
score would remain on the screen until the subject presses a 
key to bring on the next set. Thus, the subject could take 
as long as he needed for the training but could only view one 


case at a time. 


H. Experimental phase 
l. Experimental procedure common to all subjects 

The experiment was conducted over three blocks of 
twenty-six cases each. The order in which these blocks were 
presented to the subject was randomized in order to discount 
the effect of order effects. Appendix A contains details of 
the instructions given to each of the subjects at the 
beginning and end of the experiment. Each block required the 
subject to perform a repetition of one type of task. Each 
subject would sit at the terminal and read the introductory 
Screen explaining how to progress through the experiment. 
After pressing a key, the subject was presented with twenty- 
six sets of three cues. The screen was split down the middle 
So that thirteen cases appear on each side. Each cue 


represents a value as described in section C.1. The subject 
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was allowed to make an overall evaluation of the candidate and 
award a score between 1 and 9. The subject may use the arrow 
keys to move from one case to another. No requirements exist 
that require the subject to evaluate the cases in order. 
2. Use of cognitive feedback 

After ten scores have been filled in by the subject, 
he can request one of the types of feedback as described in 
section 3.5. To view the options for feedback, the subject 
need only press the END key. If no feedback was required, the 
subject may return to the experiment by pressing the N key. 
The subject may request outcome feedback at any time, but may 
no longer change his scores after doing so. When all cases 
have been evaluated, the subject again presses the END key and 
then was led through a series of questions to ensure that he 
was through with the block and then the next block was 
presented. No time limit was placed on the subject. After 
completion of each block, the subject was required to fill out 


a short questionnaire. 


I. Debriefing 

The objective of the debriefing phase was to have the 
subject provide a description of his own policy and thought 
processes as he progressed through’ the experiment. 
Information provided in nate section will give insight into 
how the subjects viewed different aspects of the experiment. 


The debriefing results will provide the experimenter 
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information concerning how clear all phases of the experiment 


were to the subjects. 


J. Summary 

This chapter explained the experimental environment. Of 
special interest in this chapter are the following points: 
(1) The experiment included four different groups of fifteen 
subjects each. Within each of these groups, the subjects were 
further differentiated by the order in which the blocks were 
presented. This is to discount order effects, should one 
block of data be significantly different from another. 
(2) The first phase of the experiment was the training phase. 
The purpose of this phase is to ensure that all subjects enter 
the experiment with equivalent knowledge of the task and no 
prejudices about the simulated task were carried into the 
experiment. The training consisted of seventy-two trials. 
(3) Cognitive feedback was provided (for half the subjects) 
in three different forms: subject policy information, subject 
consistency information, and committee policy information. 
(4) Subjects all performed the same task in the simulation. 
They evaluated potential job candidates based on three values: 
work experience, test scores, and interview score. Outcome 


feedback was provided for all subjects upon request. 
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IV. RESULTS 


A. Statistical Model 

The following analysis of variance (ANOVA) model suited 
for multiple Latin Squares was used to test hypothesis: 

Yikayjgmn 7 M * O, + 8, + I, + xı + o, * (Ix), + 

(Io), * (FR), t eiu where: 
U is constant, 
0 is the sequence of experimental tasks (see Chapter III, 
p.21), 
B, is the order of the task ) 2. 1۵۰ 
I, is the feedback condition, 
xX, is the experimental task (i.e., cue 1 irrelevant, cue 2 
irrelevant, etc.), 
O, is the experimental participant (or subject, m 2» 1,...,53), 
and 
ورن رده‎ 198 the experimental error term. 

This model was run for lens model indices achievement 
(R)), consistency (R,), and matching (G), and number of 
iterations (ITER) performed by the subject within a block. 
The analyses were conducted on SAS? statistical software, 
using the General Linear Models procedure. The general linear 
models procedure was chosen because the number of subjects 


assigned to each condition varied. The assignments of 
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subjects to blocks was as follows: CFB/H = 12, CFB/L = 14, 


OFB/H = 13, OFB/L = 12. 


B. Results 
l. Training 

The results of the training phase are summarized in 
table 1. The same dependent variables were analyzed in the 
training phase data. Different sets of cues were given to the 
subjects in the training phase based on assignment to task 
predictability conditions (i.e. all subjects in the high 
predictability conditions received the same cues in the 
training phase). All subjects received one block of decisions 
during the training phase. The only significant effects seen 
in the training data was a significant variance between groups 
(high/low predictability) in the training phase. Analysis of 
training phase data is displayed in Table 1. 

2. Experimental Results 

Tables 2 - 6 summarize the means and standard 
deviations for all of the independent variables for the 
experimental phase. Graphical representation of this data is 
provided for each set of data in the tables. 

a. Means and Standard Deviations 

Achievement 
Achievement is the measure of the agreement between 

the criterion values and responses of the subject over n 


observations. An inspection of the means for achievement (RA) 
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Table 1 TRAINING DATA ANALYSIS 


Class Levels Values 
GROUP 4 CFB/H CFB/L OFB/H OFB/L 
SEQ 3 123 


Number of observations in data set = 53 


Dependent Variable: RA 


Sum of Mean 
Source DF Squares Square F Value Pr > F 
Model 11 0.490 0.0446 4.07 0.0005 
Error 41 0.449 0.0109 
Crctd Total 52 0.939 
GROUP 3 0.4096 0.1365 12.48 0.0001* 
SEQ 2 0.0123 0.0062 0.56 0.5733 
GROUP * SEQ 6 0.0682 0.0113 1.04 0.4141 
Dependent Variable: G 
Model 11 0.010 0.0009 0.79 0.6478 
Error 41 0.048 0.0012 
Crctd Total 52 0.058 
GROUP 3 0.005 0.0017 1.50 0.2277 
SEQ 2 0.0017 0.0008 0.72 0.4949 
GROUP *SEQ 6 0.0032 0.0005 0.46 0.8345 
Dependent Variable: RS 
Model 11 0.0918 0.0080 0.97 0.4896 
Error 41 0.3537 0.0090 
Crctd Total 52 0.4454 
GROUP 3 0.0321 0.0107 1.24 0.3064 
SEQ 2 0.0130 0.0065 0.76 0.4763 
GROUP * SEQ 6 0.0466 0.0078 0.90 0.5036 


* — Significant at 0.05 level 


Table 2 MEANS AND STANDARD DEVIATIONS FOR ACHIEVEMENT 


Group Block 1 Block 2 Block 3 
CFB/H .841(.106) .818(.192) .864(.131) 
CFB/L .576(.207) . 607 (.137) .609(.149) 
OFB/H .768(.117) .744(.081) .709(.105) 
OFB/L .502(.169) .470(.231) .457(.183) 


* Mean(St.Dev.) 


in Table 2 show that subjects in the CFB/H condition had the 


highest degree of accuracy. The graphical representation of 
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this data, in Figure 4, shows that achievement in the 
cognitive feedback conditions improved in consecutive blocks 
while achievement declined in both outcome feedback 
conditions. This result supports H, as discussed in Chapter 
III. From the graph and the table it is also clear that for 
subjects within the same feedback condition, those in the high 
predictability condition had higher values for achievement 
than those in the low predictability condition. This supports 


H, as discussed in Chapter III. 


Achievement 


Meon Score 


0.4 
BLOCK 1 BLOCK 2 BLOCK 3 


—— CFB/H TF CFB/L —* OFB/H -E OFB/L 





Figure 4 Experimental results for achievement 


Consistency 
Consistency is defined as the multiple correlation 


between the cues and the criterion. Consistency for a subject 


e 


is calculated from estimates made by the subject in his/her 
final iteration ina given block. The values for consistency 
in Table 3 indicate that subjects in the CFB conditions were 
the most consistent in the utilization of their decision 
rules, as defined by the multiple regression equation. The 
graphical representation of this data, in Figure 5, shows that 
consistency improved in subsequent blocks in the cognitive 
feedback condition. 


Table 3 MEANS AND STANDARD DEVIATIONS FOR CONSISTENCY 


Group Block 1 Block 2 Block 3 
CFB/H .914(.086) .911(.077) .944(.032) 
CFB/L .914(.046) .885(.107) .923(.053) 
OFB/H .898(.045) .886(.049) .866(.073) 
OFB/L .867(.101) .839(.105) .840(.117) 


x Mean (St .Dev.) 


p ue ee ee ee 
The highest mean for consistency was found in the 
CFB/H Block 3 condition, while the lowest was found in the 
OFB/L Blocks 2 and 3 condition. Subjects in the CFB condition 
tended to become extremely consistent in the final block, with 
very little variation, as indicated by the small standard 
deviation. 
Matching 
Matching is defined as the correlation between model 
estimates of the subject with the model of the environment 


(committee policy). The values for matching in Table 4 
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Consistency 


Mean Score 


BLOCK 2 BLOCK 3 


—— CFB/H TI CFB/L —*—OFB/H — OFB/L 





Figure 5 Experimental results for consistency 
indicate that subjects in the high predictability had the 
greatest success in matching the committee policy with their 
own policy. Graphical representation of this data, in Figure 
6, shows that matching was best with cognitive feedback and 
high predictability conditions. The higher degree of matching 
between subject and committee in the cognitive feedback 
condition support H, as suggested in Chapter III. 

For cognitive feedback - low predictability 
conditions, matching improved with each subsequent block. It 
should be noted that although the matching index was on the 
average higher in the OFB/H condition than in the CFB/L 


condition, in the third block subjects in the CFB/L condition 
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Matching 


Meon Score 
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Figure 6 Experimental results for matching 


Table 4 MEANS AND STANDARD DEVIATIONS FOR MATCHING 


Group Block 1 Block 2 Block 3 
CFB/H .967(.040) .917(.032) .948(.120) 
CFB/L .809(.201) .832(.162) .870(.157) 
OFB/H .877(.148) .878(.075) ,869(.111) 
OFB/L :722(. 181) .692(.2713) .793(.099) 


* — Mean (St.Dev.) 


La o a a i سس‎ 
were able to surpass the matching ability of subjects in any 
of the OFB conditions. 

Iteration 

Iteration is described as the number of times that a 
subject requests feedback in a given block. The data in table 


V indicates that subjects in the cognitive feedback condition 
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ITERATIONS 


Mean Score 


5 
BLOCK 1 BLOCK 2 BLOCK 3 
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Figure 7 Experimental results for iteration 
completed each block in the fewest number of iterations. This 
observation does not indicate that the subjects in the 


cognitive feedback conditions completed the blocks in the 


shortest amount of time. The graphical representation of this 


data is in Figure 7. 


Table 5 MEANS AND STANDARD DEVIATIONS FOR ITERATION 
ee ee Eee Ee Eee GONE GOD 
Group Block 1 Block 2 Block 3 
CFB/H 11.9(7.90) 8.21(4.92) 71.93(3. 71) 
CFB/L 9.29(7.18) 6.86(4.00) 7.79(5.54) 
OFB/H 8.23 (7.63) 15.9(8.97) 4.5 (7.37) 
OFB/L 13.29.86) 18.3(10. 4) 13.5(9.28) 


* Mean(St.Dev.) 
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DIFFERENCE 


Mean Score 


0.4 
BLOCK 1 BLOCK 2 BLOCK 3 


—— CFB/H ~+ crB/L —* OFB/H —& OFB/L 


* between subject and committee welghts 





Figure 8 Experimental results for difference 


Difference 
Difference is described as the difference between the 
weights used by the subject and the committee weights. Thus, 
a small value for DIFF would indicate that the subject 
utilized similar cue weights as the committee. This value is 
smallest for subjects in the cognitive feedback condition. 
Graphical representation of this data is found in Figure 5. 
The means for achievement indicate that the main 
effects are due to groups (task condition) and subject 
(variations from one subject to another). The same is true 


for consistency and matching. 
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Table 6 MEANS AND STANDARD DEVIAITONS FOR DIFFERENCE 


Group Block 1 Block 2 Block 3 


CFB/H . 497) .193( .563(.287) .524(.250) 
CFB/L .666(.292) .613(.314) .531(.263) 
OFB/H .767(.311) . 758 (.209) .785(.278) 
OFB/L .834(.277) .863(.413) 1.14(1.29) 


f= Meant se .Dey .) 
Pee eee ee, ee ر س د ی‎ S EP Eme m mL, eee ee 
b. ANOVA Results 
The ANOVA was conducted on the same variables in 
the previous section, namely: RA, RS, ITER, G, and DIFF. The 
results of the ANOVA are summarized in tables 7-9. 


Table 7 ANOVA RESULTS FOR ACHIEVEMENT 


Sum of 
Source of Variation DF Squares F Value PE > F R-Square 
Model 62 4.75825204 3.95 0.0001* 0.718280 
GROUP 3 2.99613739 51.37 (۰ 
BLOCK 2 0.00370189 0.10 0.9093 
SEQ 2 0.08812805 2.2] 0.1092 
TASK 2 0.03695659 0.95 0.3901 
LNAME 47 1.57564399 Tor 0:01 
GROUP * BLOCK 6 0.05768413 0.49 0.8110 


* - Significant 0.05 level 


The ANOVA for accuracy (RA) showed significant main 
effects for group  (p,0.0001) and participant  (p«0.05). 
Variances in accuracy not found to be significant were between 
blocks, sequence, task, and group-block. 

ANOVA for matching (G) showed that the variance of 
matching was significant in the same categories as 


consistency, groups (0.0001) and last name (0.0196). Matching 
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did not vary significantly in any other conditions tested. 


Table 8 ANOVA RESULTS FOR ACHIEVEMENT 


Sum of 
Source of Variation DF Squares F Value Pr > F R-Square 
Model 62 0.66091699 2.84 0.0001* 0.647277 
GROUP 3 0.11970669 10.64 0.0001* 
BLOCK 2 0.00897107 1-20 0.3070 
SEQ 2 0.00463097 0.62 0.5416 
TASK 2 0.03546305 4.73 0.0110* 
LNAME 47 0. 46653530 2.65 0.00014 
GROUP * BLOCK 6 0.02560992 1.14 0.3465 


* - Significant at 0.05 level 


ANOVA for consistency showed that variance was 
Significant between groups (0.0001), task (0.011), and last 
name (0.0001) Consistency did not vary significantly between 
blocks, sequence, or group-block. 


Table 9 ANOVA RESULTS FOR MATCHING 


Sum of 

Source of Variation DF Squares F Value Pr ^E R-Square 
Model 62 2. 3 2.20 0.0002* 0.587311 
GROUP 3 0.87393415 14.81 0.0001* 

BLOCK 2 0.04132201 1.05 0.3537 

SEQ 2 0.09566424 2.43 0.0932 

TASK 2 0.08119316 2,06 0.1325 

LNAME 47 1.52604186 1.65 0.0196* 

GROUP *BLOCK 6 0.06850081 0.58 0.7450 


* - Significant at 0.05 level 


c. Scheffe’s Test 
Scheffe’s Test is a posterior test performed to 
determine if differences within a variable are significant 
enough to account for observed differences in experimental 


results. Scheffe’s test was performed between the following 
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variables and all possible combination of task conditions: 
RA, G, RS, ITER. 

In achievement, Scheffe’s Test indicated that the 
means were significantly different from each other (p < 0.05) 
between all task conditions. In matching, Scheffe’s Test 
indicated that means were significantly different (p < 0.05) 
in all conditions except when compared between  CFB/H - OFB/H 
and CFB - OFB/H. In consistency, all means were significantly 
different at the same level except when compared between CFB/H 


— CFB/L, CFB/L - OFB/H, and OFB/H - OFB/L. 
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V. CONCLUSIONS 


A. Scope 


The purpose of this chapter is to revisit the research 
topics discussed in earlier chapters in light of the results 
of this research. The limitations of this research and the 
potential impact on potential research will be included in 
this chapter. This chapter also discusses the conclusions of 
the research in terms of the majors areas of study with which 
it deals: information relevance and cognitive feedback; and 
limitations of this research. The chapter ends with a 
discussion of the relevance of this thesis to research in 


computer systems. 


B. Conclusions 
l. Previous Research 

We argued previously that while research in 
information relevance was abundant, research in finding means 
of reducing the effects of irrelevant information is an 
important next step. In addition to this research, an example 
of such an effort is the research done by Gaeth and Shanteau 
(1984). 

The results of the research performed by Gaeth and 


Shanteau is particularly of importance in terms of improving 
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decision quality in environments where irrelevant information 
is present. In their research, training was found to be an 
effective measure in improving experienced decision makers 
decision quality, by reducing the effects of irrelevant 
information. Similarly, in this research, the use of 
cognitive feedback was shown to be effective as well in 
improving decision quality when irrelevant information was 
present. 
2. Research Question 

The question posed by this study was: given that 
irrelevant information serves as a detriment to decision 
quality, can the use of cognitive feedback reduce the adverse 


effects of this detriment. 


C. Findings of the study 

Lens Model Indices 

Decision quality was operationalized in terms of the 
following three lens model indices: achievement, consistency, 


and matching. 


e Achievement: Achievement was shown to be the highest in 
conditions where subjects in the high predictability task 
environment received cognitive feedback. Achievement 


improved in subsequent blocks in the cognitive feedback 
condition while declining in the outcome feedback 
condition. 


e Matching: Subjects receiving cognitive feedback recorded 
higher matching index than those receiving only outcome 
feedback. 


e Consistency: Consistency was highest for subjects in the 
cognitive feedback conditions. Further, for subjects in 
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the cognitive feedback condition consistency improved in 

subsequent blocks. 

D. Application of Cognitive Feedback 

Brehmer has been a major contributor to research 
concerning cognitive feedback. His work emphasizes the 
importance of cognitive feedback to the point of dismissing 
outcome feedback entirely. Only cognitive feedback can give 
a person insight into relations between variables. Further, 
only cognitive feedback is of use in describing subject 
consistency. 

In support of Brehmer, Doherty and Balzer state that the 
human learning process is imperfect - "we need help" (Doherty 
and Balzer, 1988) Their research also points out the strong 
link between consistency and cognitive feedback. 

Cognitive feedback can be operationalized in three forms 
(discussed in chapter two): task information, cognitive 
information, and functional validity information. Various 
research on cognitive feedback has stressed each of these as 
important in different instances. While there has been some 
agreement in the research, it is clear that future research 
should focus on the distinctions between these forms of 
cognitive feedback and the importance of each. 

The use of technology in the application of cognitive 
feedback is an exciting enterprise with potential for enormous 


growth. Personal decision support systems designed to assist 


48 


the user in decisions such as major purchases or solving 
interpersonal problems are within the realm of possibility in 
the near future. Providing the user with insight into his or 
her decision processes, with the promise of improving decision 


quality, is appealing to designers in a variety of areas. 


E. Limitations and Recommendations for Further Research 
It is in the best interest of the experimenter to control 
and limit the scope an experiment in order to reduce the 
chance of numerous interaction effects or effects which cannot 
be explained by manipulation of the dependent variables. The 
loss associated with the gains of control of scope is that 
only a small part of a large picture may be examined at one 
time. The issues addressed in this section are the rest of 
the picture - those issues which bear some significance to the 
research topics but extend beyond the scope of this research 
l. Information Relevance 
From previous research on information relevance is 
Clear as to the adverse effects associated with irrelevant 
information. Other aspects of information relevance are less 
clear from the research completed to date. Further research 
should direct emphasis upon the following points: 
* Given that subjects in more predictable task environments 
are less affected by irrelevant information, where and why 
is this distinction seen? Why is cognitive feedback more 


effective in more predictable environments. 


e How does an individual determine which information is 
relevant and which is not? Do methods for sorting 
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information based on relevance increase in complexity as 
the information itself increases in complexity? 


e Overcoming the adverse effects of irrelevant information 
in decision making tasks. 


e The impact of training in improving task performance in 
task environments where irrelevant information is present. 


2. Cognitive Feedback 

One limitation to experimental research on cognitions 
and policy formation is that an experimental setting leads 
subjects to form policies and cognitions differently than in 
the real world. One example of such is knowledge acquisition. 
In this research, for example, the subject is given three cues 
from which he must make a decision. The subject has a priori 
knowledge that he will be given sets of cues. He is not 
expected to do anything to get this information. This aspect 
of the experimental setting is different to the way in which 
individuals gain knowledge in the real world. In other words, 
an experimental subject may be led to know that he or she is 
expected to form a policy, and may even be led as to what form 
of policy is expected. 

Another limitation of the study of human cognitive 
processes stems from the use of the Lens Model. The Lens 
Model, while an accepted model for studying human behavior, is 
rigid and limits the results to the confines of a finite 
number of coefficients. 

Cognitive feedback studies thus far (including this 


research) have not explored how decision makers use cognitive 
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feedback (perhaps because of the difficulty foreseen in 
operationalization). For example, when and why does a 
decision maker request feedback? Is feedback used to confirm 
an idea more effective than feedback used initially? What 
type of decision makers benefit the most from the different 
types of feedback? 

An area for further research is the use of cognitve 
feedback in conjunction with training to assist people to 
detect irrelevant information. The results of the work done 
with training discussed in this thesis indicates that there is 


great promise in this venture. 


F. Relevance to Research in Computer Systems 

One main thrust of research in computer systems is in 
decision support systems. The technology in decision support 
systems has reached a point where the competition is no longer 
between the capabilities of the hardware or the power of the 
coding. The limiting factor in most interactive software is 
the user. Thus, the area of the most potential for growth or 
additional understanding is in putting more power in the hands 
of the user. 

Conversely, power can be taken from the hands of the user 
of a decision support system through the presence of 
irrelevant information (supported by this research). The use 
of cognitive feedback shows great promise in the field of 


decision support systems. Fundamental to this is a greater 
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understanding of the human cognitive process. 


G. Summary 

The intent of this research has been to study decision 
making across different task situations. Through the 
employment of cognitive feedback, this study has shown that 
human decision making processes may be enhanced by simple 
decision aids. This result contributes to the knowledge base 
which deals with cognitive processes and decision making. A 
great deal is still unknown about the way humans make 
decisions. However, from this research and others, advances 
can be made in design technology for computer based decision 


tools. 
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APPENDIX 


The following appendix is the instruction set given to 
subject to guide them in completion of the experiment. The 
instruction set contained additional information for subjects 
in the cognitive feedback conditions. This additional 


information will be in bold. 


53 


YOUR NAME 
SMC NO. 


DESCRIPTION OF THE SIMULATION 


S This simulation seeks to gain an understanding of how 
decision makers use information for making decisions. To 
investigate the use of information, we are asking a number of 
subjects to participate in a variety of business simulations. 
The task in this simulation involves screening applicants for 
a set of positions in a large software company. You are 
required to make decisions based on data presented to you. 
The simulation has three parts. 


Part one is the training session. The objective here is to 
give you some practice on how to make decisions in this task. 


Part two is the main, or, experimental phase. In this phase, 
you make decisions based on an actual set of cases. 


Part three is the debriefing phase. 


x You will be given a set of instructions at the beginning 
of each part. 


* A brief questionnaire is to be filled out at the end of 
each part. 


* The simulation will be run on a computer terminal. At 
all times, the bottom row of the screen will prompt you about 
what to do next. 


x Please follow the guidelines strictly. The system 
prompts, along with instructions in this booklet, will guide 
you at every stage. 


x When in doubt, ask the experimenter. 


xxx please turn over for further instructions **x 
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PART I: TRAINING 


INSTRUCTIONS 
* A large software company periodically recruits personnel 
for a variety of positions. The hiring decisions are made 


from a large pool of applicants. Because the number of 
applicants is large, the task of hiring is split into two 
stages: 


First, a committee made up of experts from the human resource 
division screens the candidates. 


The results of the screening exercise are sent to individual 
departments. These departments then conduct further 
interviews and make the final hiring decisions. 


X The simulation focuses on the first part of the hiring 

task, i.e., screening candidates. In the training phase of 

the simulation, the committee of experts would like to train 
you, So that you can perform the screening task in the same 

way as they do. The objective of the training is to ensure 

that your decisions match decisions they would have made. 


E The committee of experts uses the following three 
variables in screening a particular candidate: 


Variable 1: Experience relevant to the position. 
Variable 2: General abilities test score. 
Variable 3: Interviewer rating of management abilities. 


Decisions you make must also be based on these three variables 
only. 


x Each variable is presented to the committee (and 
therefore, to you) as a range of values between 1-9, where the 
values imply the following: 


1 و ي ڪڪ و‎ 
Very Very 
Poor Good 
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You will be given profiles of several applicants. The 
profile of each applicant will be presented as a set of values 
on the three variables described above. This simulation 
requires you to rate the candidate on an overall basis, ona 
scale of 1-9, wherein: 


] fe 9س‎ 
Very Very 
Poor Good 


For Example, 


Variable 1 Variable 2 Variable 3 Dec. Maker’s 
(given) (given) (given) Score 
9 6 8 7 
4 8 9 6 
? 
? 
? 
etc. 


For the first candidate, Variable 1 had a 9, Variable 2 had a 
6, and Variable 3 an 8. The decision maker gave an overall 
score of 7 for the candidate. And so on. (By the way, the 
scores given above are just random, and not examples of actual 
values the committee would have given). 


* In the training phase, you will be given several cases 

(1.e., candidates), one at a time. After you enter a score on 
each case, the system will provide you with the actual value 
given by the committee (on the right hand side of your score). 


Your task is to predict the committee values as closely as 
possible. 


x Your first predictions will probably be guesses. 

However, as you work through the cases, you will be able to 
learn the relationships between the variables and the actual 
committee scores. You should try to come as close as possible 
to the actual committee scores. The task has been arranged so 
that you can learn to make predictions with moderate accuracy. 


As in the real world, though, predictions (of committee 
judgments) cannot be perfectly accurate all the time. 


x After completing the task, please answer the questions 
that follow immediately. 
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s Be sure you understand the instructions. When in doubt, 
ask the experimenter. 


R^ You are now ready to proceed with the task از از بر‎ 
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Questions to be answered after completing part 1 


1x 


Describe (in words, equation, etc) what decision rule you 


followed in making your estimates. 


2 Distribute 100 points among the three variables you used 
for reaching your overall estimate - in accordance with the 
importance you assigned them (be sure the total adds up to 
100). 

Variablel: 

Variable2: 

Variable3: 

TOTAL: (out of 100) 
Se How clear were the instructions regarding the task in 
this part of the simulation? 

1 2 3 4 5 6 7 8 9 

Not at all Very 

Clear Clear 
4. Please try to describe the thinking process you went 


through in making your predictions: 


kK kK Kk 


END OF PART 1 E 
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PART 2: your decisions 


INSTRUCTIONS 


s You are about to start part two of the simulation. In 
this part, you will make decisions about some actual 
applicants (i.e, cases drawn from resumes of real-life 
applicants). The task is the same as in part 1: rating 
applicants on a scale of 1-9. 


- You will make decisions on screening applicants for three 
different job positions, one position at a time. There will, 
therefore, be three different blocks of cases: one block for 
each position. 


E Descriptions of the individual job positions are not 
presented to you, so as not to bias your decisions (remember: 
your task is to emulate the committee, i.e, make decisions the 
committee would have made). However, you do need to know that 
the committee may (or may not) use somewhat different decision 
rules for different positions. 


E Since the task is for real life applicants, the system 
will provide you with decision feedback to help you make 
better decisions (i.e., to be as close to the committee scores 
as possible). The next few pages explain what decision 
feedback is and how you can use it. 


* For each block (i.e., each job position), the general 
sequence is as follows: 


dos You are given a set of 26 cases. 
2 You make estimates on these cases, in any order you like. 


The up and down arrows can be used for moving around the score 
windows (you can change your scores by simply typing over the 
old score). 


3. If you press the END key at any point, the system will 
provide you with decision feedback (however, you need to enter 
at least 10 scores in order to get such feedback). If, after 
receiving decision feedback, you would like to change any 
score, you can do so. 
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4. Once you have entered a score, you can get information 
about the actual value (i.e., the committee decision) at any 
time, also by pressing the END key. However, once you have 
received information about the actual value for a candidate, 
you cannot change your score for that candidate. 


Once you have completed a block (i.e., decided on all 26‏ بو 
scores), press the HOME key to move to the next block. At the‏ 
end of each block, please answer the questions for that block‏ 
on the questionnaire.‏ 


* At every stage, the bottom row of the screen will provide 
you with instructions about what to do next. 


* The next few pages explain decision feedback. 


x If you have a question, ask the experimenter. 
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Decision Feedback 


* What is Decision Feedback? 


Decision feedback is diagnostic information provided by 
the system on your decision processes and that of the 
committee. You can access this feedback when making your 
decisions about candidates. (In order to compute the 
information, however, the system needs at least 10 scores from 
you). By accessing such feedback, decision makers can derive 
better insight into their decisions processes. This enables 
them to revise and improve their decisions (or scores, in this 
case) through a what-if mode of analysis. 


* How do I use Decision Feedback in Making Decisions? 


Typically, you the decision maker, would use the feedback as 
follows: 


l. Make some tentative decisions (i.e., candidate scores). 
2. Ask the system for feedback. 
3. Refine your decisions accordingly. 


You may do this 1-3 sequence within a block, as many times 
(and with as many scores) as you wish. 
* Types of Decision Feedback 


The system will provide you with the following 5 types of 
feedback: 


E Information on your decision rule: 

Decision makers are sometimes unable to specify precisely, a 
particular decision rule (in this case, weights assigned to 
variables). The system will track the weights you are using 
(in formulating your scores), and will display them through a 
stacked-bar chart in figure 1. 


SEE FIGURE 2 CHAPTER III PAGE 26 
How do I use it? 
l. | Make sure the weights displayed are actually the ones you 


want applied. 
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24 If not, revise your scores, and see how the weights 
change. 


3. Iterate between 1-2 till the system shows weights you 
actually want applied. 


2. Information on your consistency: 


Sometimes, after decision makers have specified their decision 
rule, they are unable to apply them consistently. The system 
will calculate the scores you would have given had you been 
completely consistent with your decision rule. 


How do I use it? 


Ta! Check your scores against the consistency scores. 
2. Revise your scores if you need or wish to. 
3: Iterate between 1-2 till your scores match with or are 


close to the consistency scores. 


d. Information on the committee's decision rule: 


Instead of trying to figure out from several examples what 
rule the committee is using, it is more effective if 


SEE FIGURE 2 CHAPTER III PAGE 26 

How do I use it? 

1. Use the feedback to get an idea of what decision rule the 
committee has been following. 

4. Information on the committee’s decision rule and yours 
This is actually a combination of feedback 1 and feedback 3. 
It enables you to compare your decision rule with that of the 


committee and thereby emulate the committee better. 


SEE FIGURE 3 CHAPTER III PAGE 27 
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How do I use it? 


- Check weights you have given versus weights given by the 
committee. 

2. Revise your scores if you need to. 

3: Iterate between 1-2 till your weights match with or are 


close to the committee weights. 


5. Information on your decision rule and consistency 

This is actually a combination of feedback 1 and feedback 2. 
The idea here is to let you revise your weights without losing 
your consistency at the same time. 


How do I use it? 


1. Make sure the weights displayed are actually the ones you 
want applied. 


2. If not, revise your scores, and see how the weights 
change. 

3. Check your scores against the consistency scores. 
2 Revise your scores if you need or wish to. 

AKR You are now ready to proceed with the task iiio: 
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Questions to be answered after completing BLOCK 1 


[e Describe (in words, equation, etc) what decision rule you 
followed in making your own estimates (in this block)? 


p Distribute 100 points among the three variables you used 
for reaching your overall estimate - in accordance with the 
importance you assigned them (be sure the total adds up to 
100). 


Variablel: 
Variable2: 
Variable3: 


SE How do you think the committee weighted the three 
variables in this block? In other words, distribute 100 
points among the three variables according to how you think 
the committee did. 


Variablel: 
Variable2: 
Variable3: 


4. In this block, did you request decision feedback at any 
time from the system (Y/N)? 


5: If YES, try to describe how you used decision feedback in 
making your decisions. 


NUS PLEASE PROCEED TO BLOCK 2 XX 
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Questions to be answered after completing BLOCK 2 


1. Describe (in words, equation, etc) what decision rule you 
followed in making your own estimates (in this block)? 


2. Distribute 100 points among the three variables you used 
for reaching your overall estimate - in accordance with the 
importance you assigned them (be sure the total adds up to 
100). 


Variablel: 
Variable2: 
Variable3: 


e How do you think the committee weighted the three 
variables in this block? In other words, distribute 100 
points among the three variables according to how you think 
the committee did. 


Variablel: 
Variable2: 
Variable3: 


4. In this block, did you request decision feedback at any 
time from the system (Y/N)? 


5n If YES, try to describe how you used decision feedback in 
making your decisions. 














PLEASE PROCEED TO BLOCK 3 x‏ و 
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Questions to be answered after completing BLOCK 3 


1 Describe (in words, equation, etc) what decision rule you 
followed in making your own estimates (in this block)? 


2. Distribute 100 points among the three variables you used 
for reaching your overall estimate - in accordance with the 
importance you assigned them (be sure the total adds up to 
100). 


Variablel: 
Variable2: 
Variable3: 


3s How do you think the committee weighted the three 
variables in this block? In other words, distribute 100 
points among the three variables according to how you think 
the committee did. 


Variablel: 
Variable2: 
Variable3: 


4. In this block, did you request decision feedback at any 
time from the system (Y/N)? 


5 If YES, try to describe how you used decision feedback in 
making your decisions. 





X KX PLEASE turn over for further instructions  *** 
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Questions to be answered after completing PART 2 


ae. To what extent were the concepts of decision feedback (as 
explained by the instructions and the experimenter) clear to 
you? 


1 2 3 4 5 6 7 8 9 


Not at all Very 
Clear Clear 
2. To what extent was decision feedback helpful in improving 


your own decision? 


1 2 3 4 5 6 7 8 9 


Not at all Very 
Helpful Helpful 
2. To what extent was information about the actual committee 


scores helpful in improving your own decision? 


1 2 3 4 5 6 7 8 9 
Not at all Very 
Helpful Helpful 
4. How clear were the instructions regarding the task in 


this part of the simulation? 


1 2 3 4 5 6 7 8 9 
Not at all Very 
Clear Clear 
5 Now that you have completed the task, can you think of 


any other factor (other variables, etc) that may have 
influenced you in making your decisions? 


*** — END OF PART 2. *** 
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PART 3: debriefing 


J Have you, in the past, been associated with applicant 
screening before (Y/N)? 


27 If YES, to what extent was the task similar to this 
simulation? 
1 2 3 4 5 6 7 8 9 
Not at all Very 
Similar Similar 
3n How interesting was the task you just performed? 
1 2 3 4 5 6 7 8 9 
Not at all Very 
Interesting Interesting 
4. How realistic, in your opinion, was the task? 
1 2 3 4 5 6 7 8 d 
Not at all Very 
Realistic Realistic 


Please comment: 


Sr How serious were you in performing the task? 
1 2 3 4 5 6 7 8 9 
Not at all Very 
Serious Serious 
T. How clear were the instructions generally? 


1 2 3 4 5 6 7 8 3 
Not at all Very 
Clear Clear 
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e. How easy was the system to use? 


1 2 3 4 5 6 7 8 9 
Not at all Very 
Easy Easy 
9. Please give us some information about yourself (in 
absolute confidence. At no time will your name appear in the 
results. The data will only be used in an aggregate 


statistical sense). 
(a) Curriculum enrolled in: 
(b) Sex 
(c) <Age 


(d) Fulltime work experience 
(in years) 


(e) How long ago (in years) did 
you complete your 
undergraduate education? 


(f) How familiar are you with computers, generally? 
1 2 3 d 5 6 7 8 9 
Not at all Very 
Familiar Familiar 


(f) How many hours (per week) do you use computers? 


10. Your general comments regarding the simulation: 


*** END OF SIMULATION *** 


Thank you for your participation. 
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